feat: optimize the fetch_code function to accelerate the file download.#1216
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a global parallel download manager to speed up fetch_code, refactors fetch_code to enqueue and coordinate downloads, and applies formatting and small refactors across modules.
- Add
DownloadManagerwith a task queue, worker pool, and completion coordinator - Refactor
fetch_codeto use the download manager and wait for all downloads before proceeding - Tidy imports, remove extra whitespace, and update configuration paths and documentation
Reviewed Changes
Copilot reviewed 27 out of 29 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| vault/src/integration/vault_core.rs | Reorganized imports and cleaned up blank lines |
| scorpio/src/util/config.rs | Consolidated read_to_string into a single expression |
| scorpio/src/manager/fetch.rs | Added DownloadManager, refactored fetch_code concurrency |
| scorpio/script/run_1000_files.sh | New script for generating 1000-file test directory structure |
| scorpio/scorpio.toml | Updated default store_path and workspace to /tmp |
| scorpio/doc/perf_test.md | Extended performance test results and notes |
| context/src/lib.rs | Adjusted .await chaining formatting |
Comments suppressed due to low confidence (1)
scorpio/src/manager/fetch.rs:749
- [nitpick] Changing the public
fetch_treesignature to returnStringerrors may break existing callers expectingBox<dyn Error>. Consider using a consistent error type or updating all consumers accordingly.
pub async fn fetch_tree(path: &GPath) -> Result<Tree, String> {
| .await | ||
| .unwrap(); |
There was a problem hiding this comment.
Using unwrap() here can cause a panic if lfs_restore fails; propagate the error with ? so that fetch_code returns an Err instead of panicking.
| .await | |
| .unwrap(); | |
| .await?; |
| } | ||
| } | ||
|
|
||
| #[allow(unused)] |
There was a problem hiding this comment.
[nitpick] The #[allow(unused)] attribute suppresses all unused warnings on the fetch function; consider removing it or narrowing its scope to avoid hiding genuine issues.
| #[allow(unused)] | |
| // If this function is intended for future use or is invoked dynamically, document its purpose here. |
…e fetching performance.
优化思路：
添加一个全局的文件下载管理器，维护一个下载队列，在遍历目录的过程中，将需要下载的内容加载到队列中，此队列有10个固定协程进行下载。